Papers with Language model
T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition (2021.eacl-demos)
Copied to clipboard
| Challenge: | Language model (LM) pretraining has led to consistent improvements in many downstream tasks, including named entity recognition (NER). |
| Approach: | They propose a Python library for NER LM finetuning that facilitates cross-domain and cross-lingual generalization of LMs finetuned on NER. |
| Outcome: | The proposed library outperforms LMs trained on NERs in cross-domain and cross-lingual generalization tests on nine datasets. |
Chameleon: A Language Model Adaptation Toolkit for Automatic Speech Recognition of Conversational Speech (D19-3)
Copied to clipboard
| Challenge: | Language model adaptation (LMA) is a promising solution for conversational speech recognition systems. |
| Approach: | They propose to use language model adaptation techniques to adapt language models to conversational speech recognition. |
| Outcome: | The proposed toolkit compares state-of-the-art language model adaptation techniques in conversational speech recognition tasks. |
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment (N19-3)
Copied to clipboard
| Challenge: | #MeToo movement provides platform to narrate personal experiences of sexual harassment. |
| Approach: | They propose a three-part ULMFiT architecture to tackle text subtleties in a classification task . they propose to annotate a manually annotated real-world dataset to test their approach . |
| Outcome: | The proposed model outperforms existing models that rely on handcrafted stylistic features and is more accurate than generic models. |
Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model (2025.naacl-long)
Copied to clipboard
| Challenge: | Language model (LM) stores diverse factual knowledge in their parameters, which is learned during self-supervised training on unlabeled documents. |
| Approach: | They investigate the issue of "perplexity curse" in the continued training of language model (LM) they find that all studied LMs suffer from positional bias in the training document . |
| Outcome: | The proposed model is able to extract information from multiple questions with diverse queries. |
Neural Grammatical Error Correction with Finite State Transducers (N19-1)
Copied to clipboard
| Challenge: | Language model based GEC (LM-GEC) is a promising alternative to SMT and neural sequence-to-sequence models. |
| Approach: | They propose to use finite state transducers to improve LM-GEC by rescoring with neural language models. |
| Outcome: | The proposed model outperforms the best published results on the CoNLL-2014 test set and achieves far better relative improvements over the baselines. |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
Zero-shot Visual Question Answering with Language Model Feedback (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge-based visual question answering are based on pre-trained language models. |
| Approach: | They propose a language model guided captioning approach that leverages a pre-trained language model to generate captions for an image to help answer a visual question. |
| Outcome: | The proposed method outperforms several competing methods on the knowledge-based VQA task and achieves comparable results to a fine-tuned VLP model. |
Sigmoid Head for Quality Estimation under Language Ambiguity (2026.acl-long)
Copied to clipboard
| Challenge: | Language model (LM) probability is not reliable quality estimator, as natural language is ambiguous. |
| Approach: | They propose to train a language model (LM) probability module on top of pre-trained LMs to address these limitations. |
| Outcome: | The proposed module is an extra unembedding head with sigmoid activation to tackle the first limitation. |
CritiQ: Mining Data Quality Criteria from Human Preferences (2025.acl-long)
Copied to clipboard
Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, Tao Gui
| Challenge: | Existing methods to train language models rely on manual design, perplexity, or careful prompt engineering. |
| Approach: | They propose a method that automatically mines criteria from human preferences for data quality with only 30 human-annotated pairs and performs efficient data selection. |
| Outcome: | The proposed method improves on human-annotated test sets and shows high accuracy on code, math, and logic domains. |
Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
Towards the Law of Capacity Gap in Distilling Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Language model (LM) distillation aims at distilling knowledge in a large teacher LM to a small student one. |
| Approach: | They propose to use the law of capacity gap to distill knowledge from a large teacher to a small student model. |
| Outcome: | The proposed model outperforms other language models on a larger scale by using the law of capacity gap inducted from a preliminary study on small-scale (3B) LMs. |
Adaptively profiling models with task elicitation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language model evaluations fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. |
| Approach: | They propose a method that automatically builds new evaluations to profile model behavior. |
| Outcome: | The proposed method finds that language models fail in hundreds of tasks . it also finds that o3-mini is prone to hallucination when fabrications are repeated . |
Who’s Asking? Simulating Role-Based Questions for Conversational AI Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Language model users embed personal and social context in their questions. |
| Approach: | They propose a framework for simulating role-based questions using a taxonomy of asker roles for patients, caregivers, practitioners. |
| Outcome: | The proposed framework simulates 15,321 questions that embed each asker role’s goals, behaviors, and experiences. |
Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)
Copied to clipboard
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
| Challenge: | Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses. |
| Approach: | They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability. |
| Outcome: | The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. |